Papers with computational overheads
Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods have gained popularity for adapting pre-trained Large Language Models (LLMs) to downstream tasks. |
| Approach: | They propose a method to optimize the importance of full layers with layer-wise importance scoring by leveraging the estimated importance scores. |
| Outcome: | The proposed method is compatible with PEFT methods that operate on a per-layer basis and achieves better performance. |
Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | Developing techniques to support end-to-end speech translation is non-trivial because of the speech-text modality gap. |
| Approach: | They propose a coarse labeling approach that merges vocabulary labels via simple heuristic rules . they propose to use 256-bit truncation, division or modulo operations to regularize the encoder . |
| Outcome: | The proposed method can increase training efficiency while delivering better performance. |
Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inference (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models are highly advanced to user-requested tasks. |
| Approach: | They propose a device-server hybrid inference strategy for on-device LLM customization . they construct a pool of diverse base adapters and then blend them into a customized adapter . |
| Outcome: | The proposed method can be used on a large scale without extra training . it can be applied to large-scale LLMs without sacrificing the benefits of on-device customization. |
Lexically Constrained Neural Machine Translation with Levenshtein Transformer (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to incorporate lexical constraints in neural machine translation have been unsuccessful . |
| Approach: | They propose an algorithm that incorporates lexical constraints into neural machine translation. |
| Outcome: | The proposed method improves on English-German datasets without modification . it does not require any modification to the training procedure and can be easily applied at runtime with custom dictionaries. |
Few-shot Temporal Pruning Accelerates Diffusion Models for Text Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing acceleration methods for text generation ignore the importance of the distribution of sampling steps, resulting in slow sampling rates. |
| Approach: | They propose a technique to accelerate diffusion models for text generation without additional training by using a Bayesian optimization approach. |
| Outcome: | The proposed technique achieves 400x acceleration even with minimal sampling steps after down to less than 1 minute of optimization yielding a competitive performance even with minimum sampling steps. |